When are probabilistic programs probably computationally tractable?

We study the computational complexity of Bayesian inference through the lens of simulating probabilistic programs. Our approach is grounded in new conceptual hypotheses about some intrinsic drivers of computational complexity, drawn from the experience of computational Bayesian statisticians. We sketch a research program to address two issues, via a combination of theory and experiments: (a) What conditions suffice for Bayesian inference by posterior simulation to be computationally efficient? (b) How should probabilistic programmers write their probabilistic programs so as to maximize their probable tractability? The research program we articulate, if successful, may help to explain the gap between the unreasonable effectiveness of some simple, widely-used sampling algorithms and the classical study of reductions to Bayes from hard problems of deduction, arithmetic, and optimization. Predicting the runtime of programs in knowledge-based systems, where executing a program involves applying a general-purpose reasoning algorithm to some knowledge representation, has always been challenging. In logic programming, programmers were taught to guess the tractability of their programs by acquiring intuition from direct experience implementing complex general reasoning algorithms as well as a large body of special cases. The practice of computational Bayesian statistics is quite similar: even when using general purpose tools like BUGS [LTBS00], experts rely on intuitions derived from lengthy experience implementing specialized samplers for specific model classes [GRS96]. If probabilistic programming (and computational Bayesian statistics, more generally) is to spread beyond the small community of Bayesian statisticians and inference engine designers, we need to dramatically improve this learning curve. In particular, we need to empower domain experts to build and perform inference over rich, realistic models without an extended education in Monte Carlo. In service of the eventual goal of producing a simple, teachable methodology, we focus on the following two tasks: (a) Identify broadly applicable sufficient conditions under which probabilistic programs (including, but not limited to, Bayesian inference problems represented as probabilistic programs) are computationally tractable. (b) Develop a clear, communicable design methodology for probabilistic programs, combining these sufficient conditions and intuitions derived from experiments, that lets probabilistic programmers solve problems in terms of programs that will probably terminate quickly enough and with acceptable accuracy. At minimum, this methodology should help probabilistic programmers detect and resolve situations where it seems unlikely that rapid, accurate termination is feasible. We have been attempting to address these issues, and here propose a research program that bridges the gap between the very different intuitions from deterministic computational complexity theory and the experience of computational Bayesian statisticians. The sketch we present here includes new theory and experiments as well as a reinterpretation of some recently developed theory. In this note, we present three claims, along with some of the theoretical and empirical evidence we have gathered so far that lends support to each. We briefly collect the claims here: